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AMENDMENTS TO THE CLAIMS 

Prior to the present communication, claims 1-66 were pending in the subject 
application. Each of claims 1-21, 23-43, and 45-59 has been amended herein, while no claims 
have been cancelled or added. Thus, claims 1-66 remain pending. It is respectfully submitted 
that no new matter has been added by way of the present amendments. All claims currently 
pending and under consideration in the present application are shown below. This listing of 
claims will replace all prior versions, and listings, of claims in the application: 
Listing of Claims: 

1 . (Currently Amended) A method for automating the optimization of search 
results displayed in a search Web page, the method comprising: 

providing receiving search results to users of a provided from a search 
engine to a user according to a search term, wherein the search results are selected 
and ranked by a relevance schema ; 

collecting data that represents a performance o f a search result each of the 
received search results , the collected data quantifying interactions of various users 
with the search results, the collected data originating from at least one of a 
plurality of various sources having various types of valuations of performance 
data that includes implicit data collected by quantifying the user's interactions 
with the search results ; 

combining normalizing the collected data by a normalization procedure 
that describes the user's interactions with the search results in accordance with a 
relative importance of the source of the data that generates performance data for 
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each of the search results, wherein the performance data reflects behavior of the 
various users, and wherein the normalization procedure comprises: 

(1) aggregating the collected data from the various sources; 

(2) compiling various types of valuations, associated with the 
aggregated data, into common measurements; 

(3) weighting the common measurements based on relative 
importance of the various sources, wherein the relative performance is a 
reflection of the value of each of the various sources as a predictor of 
relevance of the search results; and 

(4) normalizing the weighted common measurements to determine 
performance data associated with each of the search results such that the 
search results are comparable against each other and search results 
indicated by expected performance data; 

determining whether the expected performance data is below the 
normalized performance data by comparing the normalized performance data to 
an -the expected performance data for each of the search results; 

when the normalized performance data of a search result of the search 
results is below the expected performance data associated therewith, identifying 
the search result as underperforming and diagnosing at least one possible cause 
for an the underperforming search result from based on t he— results of the 
comparison between the normalized performance data and the expected 
performance data; and 



Application No. 10/805,731 

File Date 03/22/2004 

Reply to Office Action of 02/12/2008 

Response Filed 05/16/2008 



MS #306409 .01/MFCP.1403 15 



updating the relevance schema based on the diagnosis such that adjusting 



an-operation of a search engine that produced provided the search results is 



adjuste d in accordance with the diagnosis to improve relevance of t he- sub sequent 



search results perfc 



:, the adjustment operativ 



give greater significance 



to collected perfc 



data from at least 



of the 



when normalizing 



2. 



(Currently Amended) The method of Claim 1 , wherein the at least 



-ef 



the plurality of various sources of performance data includes one of an implicit performance 
data, an explicit performance data, a human-judged performance data, a relevance verification 
data, and a sample test data. 

3. (Currently Amended) The method of Claim 1, wherein the normalization 
procedure normalizing the collected data in accordance with a relative importance of the source 
of the data includes giving greater weight to the collected data that describes the-a_user's 
interactions that receives the search results in relation to the various users with the search results 
and combining the data to reflect the relative importance of the source from which the data 
originated . 

4. (Currently Amended) The method of Claim 2, wherein comparing the 
normalized performance data te -against the expected performance data includes determining that 
identifying the search result is-as underperforming when the normalized performance data for the 
search result is below a quantified threshold below the expected performance data . 
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5. (Currently Amended) The method of Claim 2, wherein the most important 
source of data is implicit performance data, and normalizing the collected data the normalization 
procedure includes giving the implicit performance data greater weight when combining the 
collected data. 

6. (Currently Amended) The method of Claim 25, wherein the implicit 
performance data is automatically captured when a use r that is provided the search results 
interacts with the search results. 

7. (Currently Amended) The method of Claim 6, wherein the implicit 
performance data includes at least one of whether the user clicked on the result, a location of the 
result when the user clicked the result, and a length of time that the user browsed the result. 

8. (Currently Amended) The method of Claim 7, wherein the implicit 
performance data further identifies an operation that the user performed on the result, including 
at least one of editing, e-mailing, printing, bookmarking, and copying. 

9. (Currently Amended) The method of Claim 6, wherein the implicit 
performance data includes captured data that has been aggregated across multiple interactions 
with the search result. 

10. (Currently Amended) The method of Claim 6, wherein the implicit 
performance data includes captured data that has been aggregated across multiple users 
interacting with the search result. 
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11. (Currently Amended) The method of Claim 2, wherein the explicit 
performance data includes data obtained from at least one user responses to a search engine 
operator-generated inquiry about the search results, wherein the inquiry includes one of an on- 
line inquiry and a telephone inquiry. 

12. (Currently Amended) The method of Claim 2, wherein the human-judged 
performance data includes data obtained from a human evaluation of the search result. 

13. (Currently Amended) The method of Claim 2, wherein the relevance 
verification data includes data obtained from a verification test of the search result's relevance to 
verify whether a search result having a known relevance is still included in the search result 
generated by the search engine. 

14. (Currently Amended) The method of Claim 2, wherein the sample test 
data includes data obtained from a test of the search result's relevance performed on a sample of 
a subset of users. 

15. (Currently Amended) The method of Claim 2, wherein diagnosing at least 
one possible cause for an- the underperforming search result includes making the following 
determinations: 

determine at least one of whether the search result is no longer valid; 
determine whether the search result appears in a poor location; 
determine whether the search term that generated the search result is easily 
misspelled; and 
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determine whether a search for the search term should be constrained to a 
specific resource 

considering at least one of whether the search result is no longer valid, 
whether the search result appears in a poor location, whether a search term that 
generated the search result is easily misspelled, whether the search term is too 
broad to generate a meaningful result, and whether a search for the search term 
should be constrained to a specific resource . 

16. (Currently Amended) The method of Claim 2, wherein adjusting-an the 
operation of a- the search engine that produced provided the search results in accordance with the 
diagnosis to improve the search result performance, includes modifying the search engine's 
search schema to change the subsequent search result generated for the search term , including at 
least one of by reranking, removing, and-or replacing the search results. 

17. (Currently Amended) The method of Claim 2, wherein adjusting-an the 
operation of a-the search engine that produced provided the search results in accordance with the 
diagnosis to improve the search result performance, includes modifying the search engine's 
search schema to augment a presentation of the search results generated for the search term, 
including at least one of highlighting, animating, enlarging, and repositioning an appearance of 
the search result on a search results web page. 

18. (Currently Amended) The method of Claim 2, wherein adjusting-an the 
operation of a- the search engine that produced p rovided the search results in accordance with the 
diagnosis to improve the search result performance, includes increasing the search engine's 
spellchecker tolerance. 
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19. (Currently Amended) The method of Claim 2, wherein adjusting-an the 
operation of a-the search engine that produced provided the search results in accordance with the 
diagnosis to improve the search result performance, includes prompting the user to one of clarify 
or narrow the search term with an additional user input. 

20. (Currently Amended) The method of Claim 2, wherein adjusting-an the 
operation of a-the search engine that produced provided the search results in accordance with the 
diagnosis to improve the search result performance, includes temporarily adjusting the operation 
of the search engine, and determining whether the adjustments have actually improved the search 
result performance before permanently adjusting the operation of the search engine. 

21. (Currently Amended) The method of Claim 2, wherein adjusting-an- the 
operation of a- the search engine that produced provided the search results in accordance with the 
diagnosis to improve the search result performance, includes adjusting the operation of the 
search engine in real time. 

22. (Original) The method of Claim 21, wherein adjusting the operation of the 
search engine in real time includes intercepting the search result generated by the search engine 
and modifying the search result before the search engine displays the search result to the user. 

23. (Currently Amended) An automated search result optimization system 
implemented in a computer that provides search results to a user, comprising: 

an input component to receive an-input data representing a performance of 
a search result generated by a search engine that is executed on the computer for a 
search term, the input data originating from a plurality of various sources; 
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a data collection processor to collect the input performance data from the 
plurality of various sources wherein the input data collected includes implicit 
performance data that describes an aggregation of the user's various user 
interactions during various sessions with the search result; 

a diagnostic comparison processor to compare a — the collected input 
performance data of the search resul t, as represented by the collected performance 
data, to an expected performance of the search result; and 

a diagnostic processor to determine, based on the comparison, whether the 
collected input performance data diverges from the expected performance by a 
quantified threshold, and if so, identify the search result as underperforming and 
diagnose the underperforming search result utilizing results of the comparison to 
select from a set of predefined corrective actions, wherein one of the set of 
corrective actions is implementing an adjustment processor; and 

an- the adjustment processo r to generate an output data representing an 
action to automatically adjust an-operation of the search engine whenever the 
search result performance diverges a quantified threshold from the expected 
performance , 

wherein the adjusted operation of the search engine is to improves the 
performance of the search result by tailoring a ranking of subsequent search 
results in accordance with the collected input performance data, IT;ll and 

wherein the adjustment gives greater significance an increased weight is 
attached to the collected input performance data from one or more of the various 
sources based on the implicit performance dat a thereby influencing the automatic 
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adjustment of the operation of the search engine 




interactions with the search result . 



24. (Currently Amended) The system of Claim 23, wherein the at least one of 



the plurality of sources of input performance data includes one of an- the implicit performance 
data, an explicit performance data, a human-judged performance data, a relevance verification 
data, and a sample test data. 



process includes a normalizing process to normalize the collected data in accordance with a 
relative importance of the-a_source of the collected data within the plurality of various sources , 
and further wherein normalizing includes giving greater weight to the collected data from the 
more important sources and combining the collected data to reflect the relative importance of the 
source from which the collected data originated. 

26. (Currently Amended) The system of Claim 25, wherein the diagnostic 
processor compares the performance of the search result as represented by the normalized 
collected performance data to an- the expected performance of the search result ; and further , and 
wherein the comparison is unfavorable the search result is underperfomiing when the 
performance is lower than a -the quantified threshold value below the expected performance. 

27. (Currently Amended) The system of Claim 25, wherein the most 
important source of data is implicit performance data, and normalizing the collected input 
performance data includes giving the implicit performance data greater weigh t, in relation to a 
balance of the collected input performance data, when combining the data. 



25. (Currently Amended) The system of Claim 23, wherein the data collection 
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28. (Currently Amended) The system of Claim 27, wherein the implicit 
performance data is automatically captured when a-the user interacts with the search result. 

29. (Currently Amended) The system of Claim 28, wherein the implicit 
performance data includes at least one of whether the user clicked on the search result, a location 
of the search result when the user clicked the search result, and a length of time that the user 
browsed the search result. 

30. (Currently Amended) The system of Claim 29, wherein the implicit 
performance data further includes data representing an identification of an operation that the user 
performed on the search result, including at least one of editing, e-mailing, printing, 
bookmarking, and copying. 

31. (Currently Amended) The system of Claim 28, wherein the implicit 
performance data includes captured data that has been aggregated across multiple interactions 
with the search result. 

32. (Currently Amended) The system of Claim 28, wherein the implicit 
performance data includes captured data that has been aggregated across multiple users 
interacting with the search result. 

33. (Currently Amended) The system of Claim 24, wherein the explicit 
performance data includes data obtained from the u ser's response to a search engine operator- 
generated user z satisfaction survey about the search result. 
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34. (Currently Amended) The system of Claim 24, wherein the human-judged 
performance data includes data obtained from a human evaluation of the search result. 

35. (Currently Amended) The system of Claim 24, wherein the relevance 
verification data includes data obtained from a verification test of the search result's relevance to 
verify whether a search result having a known relevance is still included in the search result 
generated by the search engine. 

36. (Currently Amended) The system of Claim24, wherein the sample test 
data includes data obtained from a test of the search result's relevance performed on a sample of 
a subset of users. 

37. (Currently Amended) The system of Claim 24, wherein the diagnostic 
process diagnostic process is further configured to further includes diagnosing diagnose at least 
one possible reason why the performance of the underperforming search result compares 
unfavorably te— against the expected performance , wherein the reason is determined upon 
executing one or more of a set of processes comprising , including considering at least one of 
whether the search result is no longer valid, considering whether the search result appears in a 
poor location, whether a search term that generated the search result is easily misspelled, 
considering whether the search term is too broad to generate a meaningful result, and or 
considering whether a search for the search term should be constrained to a specific resource. 
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38. (Currently Amended) The system of Claim 23, wherein the adjustment 
processor is further configured to generate an output data representing an action to automatically 
adjust an operation of the search engine, includes generating output data that represents an action 
to modify the search engine's search schema, wherein the modified search schema changes the 
way the search engine generates a-search results for a search term, including at least one of 
reranking, removing, and replacing the search result. 

39. (Currently Amended) The system of Claim 23, wherein the adjustment 
process or is further configured to generate an output data representing an action to automatically 
adjust an operation of the search engine, includes generating output data that represents an action 
to modify the search engine's search schema, wherein the modified search schema augments the 
way the search engine presents the search result for a search term, including at least one of 
highlighting, animating, enlarging, and repositioning an appearance of the search result on a 
search result Web page. 

40. (Currently Amended) The system of Claim 23, wherein the adjustment 
process or is further configured to generate an output data representing an action to automatically 
adjust an operation of the search engine, includes generating output data that represents an action 
to increase the search engine's spellchecker tolerance without changing the spelling or creating a 
variation of the search term . 

41. (Currently Amended) The system of Claim 23, wherein the adjustment 
process or is further configured to generate an output data representing an action to automatically 

to prompt the user to one of clarify or narrow the search term with an additional user input. 
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42. (Currently Amended) The system of Claim 23, wherein the adjustment 
process or is further configured to generate an output data representing an action to automatically 
adjust an operation of the search engine, includes generating output data that represents an action 
to temporarily adjust the operation of the search engine, and to further determine whether the 
temporary adjustment has actually improved the search result performance before generating an 
action to permanently adjust the operation of the search engine. 

43. (Currently Amended) The system of Claim 23, wherein the adjustment 
process or is further configured to generate an output data representing an action to automatically 
adjust an operation of the search engine, includes generating output data that represents an action 
to adjust the operation of the search engine in real time. 

44. (Previously Presented) The system of Claim 43 wherein the action to 
adjust the operation of the search engine in real time includes an action to intercept the search 
result generated by the search engine and to further modify the search result before the search 
engine displays the search result to the user. 

45. (Currently Amended) One or more computer-accessible media having 
instructions stored on the media for facilitating the automated optimization of a search result in a 
search result user interface, the instructions comprising: 

providing receiving search results to users of a provided by a search 
engine in response to an inputted search term from a user ; 

collecting data that represents a performance of a- each search resul t of the 
search results from at least one of a plurality of various sources of performance 
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data that includes implicit data collected by quantifying the user's interactions of 
various users during various sessions with each of the search results; 

normalizing the collected performance data that describes the user's 
interactions with the search results in accordance with a relative importance 
associated with each of the various sources source of the performance data, 
wherein normalizing comprises formatting the performance data in a common 
measurement that is associated with each of the search results such that the search 
results are comparable against each other and search results associated with 
expected performance data ; 

comparing the normalized performance data te-an -against the expected 
performance data for the search result; 

based on the comparison, determining whether the collected performance 
data diverges from the expected performance by a quantified threshold and 
identifying a search result, of the search results, as underperforming when it is 
associated with divergent collected performance data; 

diagnosing at least one possible cause for an- the underperforming search 
result based on a result of the comparison between the normalized performance 
data and the expected performance data; and 

utilizing the at least one possible cause to select from a set of predefined 
corrective actions, wherein one of the set of corrective actions is adjusting an 
operation of the search engine adjusting an operation of a search engine that 




-in accordance with the diagnosis to improve the search 



jult perfc 



the adjustment operative to give greater signify 
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collected performance data from at least one of the sources when normalizing the 

46. (Currently Amended) The computer-accessible media of Claim 45, 
wherein the at least one of the plurality of various sources of performance data includes one of 
an- the implicit performance data, an explicit performance data, a human-judged performance 
data, a relevance verification data, and a sample test data. 

47. (Currently Amended) The computer-accessible media of Claim 45, 
wherein the instruction to normalize the collected data that describes the user's interactions with 
the search results in accordance with a relative importance of the source of the data includes 
instructions to give greater weight to the data from the more important sources of the at least one 
of the plurality of various sources and to combine the data to reflect the relative importance of 
the source from which the data originated. 

48. (Currently Amended) The computer-accessible media of Claim 45, 
wherein the instruction to compare the normalized performance data to the expected performance 
data includes an instruction to determine that the search result is underperforming when the 
normalized performance is below a the quantified threshold value below the expected 
performance. 

49. (Currently Amended) The computer-accessible media of Claim 47, 
wherein the most important source of data is the implicit performance data, and the instruction to 
normalize the collected data includes an instruction to give implicit performance data greater 
weight when combining the data. 
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50. (Currently Amended) The computer-accessible media of Claim 46, 
wherein the implicit performance data is data that is automatically captured by a search engine 
when a user interacts with a search result. 

51. (Currently Amended) The computer-accessible media of Claim 50, 
wherein the implicit performance data includes data that represents at least one of whether the 
user clicked on the result, a location of the result when the user clicked the result, and a length of 
time that the user browsed the result. 

52. (Currently Amended) The computer-accessible media of Claim 50, 
wherein the implicit performance data further identifies an operation that the user performed on 
the result, including at least one of editing, e-mailing, printing, bookmarking, and copying. 

53. (Currently Amended) The computer-accessible media of Claim 50, 
wherein the implicit performance data includes captured data that has been aggregated across 
multiple interactions with the search result. 

54. (Currently Amended) The computer-accessible media of Claim 50, 
wherein the implicit performance data includes captured data that has been aggregated across 
multiple users interacting with the search result. 

55. (Currently Amended) The computer-accessible media of Claim 46, 
wherein the explicit performance data includes data obtained from the user's response to a search 
engine operator-generated inquiry about the search result, wherein the inquiry includes one of an 
on-line inquiry and a telephone inquiry. 
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56. (Currently Amended) The computer-accessible media of Claim 46, 
wherein the human-judged performance data includes data obtained from a human evaluation of 
the search result. 

57. (Currently Amended) The computer-accessible media of Claim 46, 
wherein the relevance verification data includes data obtained from a verification test of the 
search result's relevance to verify whether a search result having a known relevance is still 
included in the search result generated by the search engine. 

58. (Currently Amended The computer-accessible media of Claim 46, wherein 
the sample test data includes data obtained from a test of the search result's relevance performed 
on a sample of a subset of users. 

59. (Currently Amended) The computer-accessible media of Claim 45, 
wherein diagnosing at least one possible cause for an underperforming search result includes 
generating an instruction to make the following determinations: 

determine at least one of whether the search result is no longer valid[[,]]; 

determine whether the search result appears in a poor location; and 

determine whether a -the search term that generated the search result is 
easily misspelled ; and 

determine whether a search for the search term should be constrained to a 
specific resource . 
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60. (Original) The computer-accessible media of Claim 45, wherein the 
instruction to adjust the operation of the search engine that produced the search result in 
accordance with the diagnosis, includes an instruction to modify the search engine's search 
schema to change the search result generated for the search term, wherein to change the search 
result includes at least one of to rerank, remove, and replace the search result. 

61. (Original) The computer-accessible media of Claim 45, wherein the 
instruction to adjust the operation of the search engine that produced the search result in 
accordance with the diagnosis, includes an instruction to modify the search engine's search 
schema to augment a presentation of the search result generated for the search term, wherein to 
augment the presentation includes at least one of to highlight, animate, enlarge, and reposition a 
display of the search result on a search result Web page. 

62. (Original) The computer-accessible media of Claim 45, wherein the 
instruction to adjust the operation of the search engine that produced the search result in 
accordance with the diagnosis, includes an instruction to increase the search engine's 
spellchecker tolerance. 

63. (Original) The computer-accessible media of Claim 45, wherein the 
instruction to adjust the operation of the search engine that produced the search result in 
accordance with the diagnosis, includes an instruction to prompt the user to one of clarify or 
narrow the search term with an additional user input. 
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64. (Original) The computer-accessible media of Claim 45, wherein the 
instruction to adjust the operation of the search engine that produced the search result in 
accordance with the diagnosis, includes an instruction to temporarily adjust the operation of the 
search engine, and an instruction to further determine whether the adjustments have actually 
improved the search result performance before performing an instruction to permanently adjust 
the operation of the search engine. 

65. (Original) The computer- accessible media of Claim 45, wherein the 
instruction to adjust the operation of the search engine that produced the search result in 
accordance with the diagnosis, includes an instruction to adjust the operation of the search 
engine in real time. 

66. (Original) The computer-accessible media of Claim 65, wherein the 
instruction to adjust the operation of the search engine in real time includes an instruction to 
intercept the search result generated by the search engine and a further instruction to modify the 
search result before the search engine displays the search result to the user. 
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